build pipeline
ML.NET Model Lifecycle with Azure DevOps CI/CD pipelines Cesar de la Torre
When you infuse AI (such as an ML.NET model) into your application, then your application lifecycle needs to be extended so it additionally embraces the'Machine Learning Model Lifecycle'. When deploying ML models to production, you need to automate the process to track, version, audit, certify and re-use every asset in your ML model lifecycle along with the end-user application lifecycle. In short, the ML model lifecycle process must be part of the application's Continuous Integration (CI) and Continuous Delivery (CD) pipelines. Let's walk through the diagram above to understand how this integration between the ML model lifecycle and the app development lifecycle can be achieved. For this common scenario, a starting assumption is that Git is used as your code repository, but it could be any other source code management platform.
Build pipelines with Pandas using "pdpipe"
Pandas is an amazing library in the Python ecosystem for data analytics and machine learning. They form the perfect bridge between the data world, where Excel/CSV files and SQL tables live, and the modeling world where Scikit-learn or TensorFlow perform their magic. A data science flow is most often a sequence of steps -- datasets must be cleaned, scaled, and validated before they can be ready to be used by that powerful machine learning algorithm. These tasks can, of course, be done with many single-step functions/methods that are offered by packages like Pandas but a more elegant way is to use a pipeline. In almost all cases, a pipeline reduces the chance of error and saves time by automating repetitive tasks.